Information Theory
This page contains resources about Information Theory in general. More specific information is included in each subfield. Subfields and Concepts See Category:Information Theory for some of its subfields. * Shannon entropy * Cross entropy / Joint entropy * Conditional entropy * Differential entropy * Information content * Mutual Information * Relative entropy / Kullback-Leibler divergence / Information gain * Entropy encoding ** Huffman coding ** Arithmetic coding * Minimum description length (MDL) principle * Minimum Message Length (MML) * Occam's Razor * Solomonoff's Theory of Inductive Inference * Kolmogorov Complexity * Principle of Maximum Entropy * Hamming distance * Hamming code * Wavelets * Information bottleneck * Shannon's Source Coding Theorem / Noiseless Coding Theorem * Neural Network Compression / Model Compression ** Nodes pruning ** Weight pruning ** Quantization of weights ** Structured Sparsity Learning ** Soft-weight sharing ** SqueezeNet Architecture ** Variational Dropout * Coding Theory ** Data Compression / Source Coding *** Lossy Compression *** Lossless Compression **** Probabilistic Data Compression ***** Prediction by partial matching (PPM) ***** Sequence Memoizer ***** Bayesian Networks ** Error Correction / Channel Coding ** Cryptographic Coding ** Line Coding ** Sparse Coding ** Deep Compression ** Bayesian Compression ** Dynamic Network Surgery * Applications ** Cryptography ** Communication Systems ** Machine Learning / Pattern Recognition ** Statistical Learning Theory ** Estimation Theory / Statistical Signal Processing ** Bayesian Inference Online Courses Video Lectures *Information Theory and Coding by S.N.Merchant *Information Theory, Pattern Recognition, and Neural Networks by David MacKay *Information Theory by Raymond W. Yeung *Probability, Information Theory and Bayesian Inference by Joaquin Quiñonero Candela *Information, Entropy and Computation by Paul Penfield and Seth Lloyd (Notes) Lecture Notes *Information Theory by Tsachy Weissman *Information Theory by Muriel Médard *Information Theory by Yao Xie *Advanced Topics in Information Theory by Stefan M. Moser *A Short Course in Information Theory by David J.C. MacKay *Information Theory by Radford Neal *Information theory in computer science by Mark Braverman *Information Theory in Computer Science by Anup Rao *Information Theory and its applications in theory of computation by Venkatesan Guruswami and Mahdi Cheraghchi *Information Theory by Iain Murray *Information Theory by Cong Ling *Network Information Theory by Abbas El Gamal Books See also Textbooks. Introductory * Moser, S. M., & Chen, P. N. (2012). A Student's Guide to Coding and Information Theory. Cambridge University Press. * Gray, R. M. (2011). Entropy and information. In Entropy and Information Theory. Springer New York. * Yeung, R. W. (2008). Information Theory and Network Coding. Springer. * Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. John Wiley & Sons. Specialized * El Gamal, A., & Kim, Y. H. (2011). Network Information Theory. Cambridge University Press. * Merhav, N. (2010). Lecture Notes on Information Theory and Statistical Physics. Foundations and Trends® in Communications and Information Theory 6(1-2): 1-212 . * Anderson, D. R. (2008). "Chapter 3: Information Theory and Entropy". Model Based Inference in the Life Sciences. Springer New York. * MacKay, D. J. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press. Scholarly Articles * Louizos, C., Ullrich, K., & Welling, M. (2017). Bayesian Compression for Deep Learning. In Advances in Neural Information Processing Systems (pp. 3290-3300). * Ullrich, K., Meeds, E., & Welling, M. (2017). Soft Weight-Sharing for Neural Network Compression. arXiv preprint arXiv:1702.04008. * Molchanov, D., Ashukha, A., & Vetrov, D. (2017). Variational Dropout Sparsifies Deep Neural Networks. arXiv preprint arXiv:1701.05369. * Wen, W., Wu, C., Wang, Y., Chen, Y., & Li, H. (2016). Learning Structured Sparsity in Deep Neural Networks. In Advances in Neural Information Processing Systems (pp. 2074-2082). * Han, S., Mao, H., & Dally, W. J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint arXiv:1510.00149. * Steinruecken, C. (2014). Lossless Data Compression. PhD Diss., University of Cambridge. * Alajaji, F., & Chen, P. N. (2013). Lecture Notes in Information Theory: Part I. * Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information Bottleneck Method. arXiv preprint physics/0004057. Software * Information Theory Toolbox - MATLAB * Octave-Information_Theory - Octave * Module pyentropy - Python * List of Compression Algorithms - Python * Module PyNLPl.statistics - Python * Information Theory and Signal Processing Library (libit) - C * NSB Entropy Estimation See also * Machine Learning * Signal Processing * Stochastic Processes * Probability Theory * Statistical Learning Theory Other Resources * Information Theory - Google Scholar Metrics (Top Publications) *Video Tutorials - Youtube channel of 'Mathematical Monk' *Soft weight-sharing for Neural Network Compression - Github *Bayesian Compression for Deep Learning - Github *Dynamic Network Surgery - Github *Software by IEEE Information Theory Society *Programming notes for Information Theory *Information Theory by Wikiversity *Information Theory - Notebooks Category:Information Theory